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ABSTRACT 


This study examines the accuracy of an Army helicopter 
pilot workload measuring model called the Task Loading Model. 
The model is a submodel of the Army-NASA Aircrew/Aircraft 
Integration Program's Man-Machine Integration Design and 
Analysis System. The model's workload level output was 
correlated with the subjective workload measurements of 
several groups of pilots evaluating a variety of flight tasks. 
Seventy-one Army aviators completed surveys requiring scaled 
ratings and paired comparisons of workload related to common 
flight tasks conducted during typical missions. Their 
responses were examined for internal consistency and pooling 
by means of nonparametric tests. Aviator-supplied data was 
found to be robust and reliable. Pooled response data was 
correlated with model-generated data to determine the accuracy 
of the model. Results of this study show that the Task 
Loading Model is presently inadequate, but displays promising 


trends and should be further refined. 


pigs 


Il. 


TABLE OF CONTENTS 


INTRODUCTION (20250. 5 Soares ooo cee os | cect tan - 
A. BACKGROUND ..2..27-. so eres Terr rrrreeT OO } 
1. Operator Error and the DeSign Process ........ 1 
2. Army-NASA Design Tool: MIDAS ............+. ee 2 
3. MIDAS Task Loading Model... ....2... cos s 4 
Be TIM PROBLEMS e .<.csesns os eee e cies eaehenenste tous ere ; 3 
C.. PURPOSE AND SCOPE. wees ese eee ce he 00 6 6 6 6 0 + 6 enn 
METHODOLOGY hs see re arene Teer eee eee eee es OO 
A. STUDY OVERVIEW 2 .cce ce ceetie oo eo eeielie see clei tence tne 7 
B. SCENARIOS AND TASKS errr es cic ienel cle ole cc cMementes. «(acum 
1. FLIght Scenarrve DevelopmentGs.. ... scans eer 7 
2. Task. S€lect1 On weer. 1. ar 8 
C. SUBJECTIVE WORKLOAD MEASUREMENT TECHNIQUES ...... 12 
1. Selection of Measurement Techniques ......... 12 
2. Rating SCal Gsim.n. .1...0ewer Pree ee i ree Le 
3. Thurstone’s Method 2.39%... . . . seen i 
D. SURVEY CONSTRUCTION -. c= De Oc COMME a eo ee CL 
1. Survey Portate.n... .. ee. ere eee er ree LS 
2s. SULVCY “CONUCEM Ct. oe tate iss ete ee oe soos ers ee os oem 
3. Minimizing the Social-Desirability Response . 16 
Ew. SURVEY ADMINISTRATION cic. ceive cece: « occ « 6 ces pemeemns soe LZ 


Lv 


gp Cuenta ae siete) ofepo(s (=) seta wearer ee nee ne eee ar ee ere er 1g 


2. Survey Administration Procedures ............ 18 
eA IY 6 Poe Or wot y (DA RA ao ds 5 606i Mateos soe io Nes) oe) oe ew nelle oo es 19 
FM BYR See Ni 2 Bp Ge Ee ers Se Gl WU ae ee eae eae or ee eee arr ee yer vers area’ Lo 
Be  PITOreBACKGROUND: [INFORMATION micccci 5 ee eus fe es so ese eles 20 
C. DATA TRANSFORMATION AND SUMMARIZATION ........... Ze 
Beeb LOho. or WORKLOAD. DISTRIBUTION Cis a4s oe 8 6669 oo ws 25 
BN ON ee Arr Ce SS Saves ice ore te ei el ee teltene? she) oe: o\-e ioe. ow Sytem eanens Zi 
deen she ChIVGdy ce iter saviety ct ratee rates, Oo e4aii5 B25 Sice doe ww. Oy ae 28 

2 ope armanonank COPrelat TONS = sa: cies ccs 1s wists © «Ses 30 

Be Ww LCOxORn SIONeG RANK Gest yi tac. <6 woe wie S eke sere s 32 

EP. FOObED WORKLOAD RESPONSES notes sele ie. & 4 oo ee @ € 6 4 coun ote 34 


TV. 


G. CORRELATION OF RATINGS WITH PAIRED-COMPARISONS .. 38 
COMPARISON OF MIDAS MODEL DATA WITH STUDY DATA ...... 42 
A. TASK LOADING MODEL WORKLOAD VALUES .............. 42 
B. RANK CORRELATION OF MODEL DATA WITH SURVEY DATA . 44 


C. GRAPHIC ANALYSIS OF MODEL DATA AND SURVEY DATA .. 48 


1. Data Selection for Further Analysis ......... 48 
22P  RESUIESSOT CANIS 1S x.suehe eens tumours Siu sissies, ee as 49 
CONCLUSIONS AND ORE COMMENDA TER ONS senveseccs a sets auc. c dtu: steer emacs 55 
es Ke PIAS ees IEC IN Oo serous Vorseementen eet er euls tice: seman Se latroneuis: AE ec) A RNIiet Gxte) Glee aes a5 
alarms ts (AN (ero il eae lay cht ae enue ay NeRp Pe en eee 55 


2. Army Helicopter Pilot Workload Perceptions .. 56 


Pe ere Ce ee eo RR CO SK Ga oc oY 

C. SUGGESTIONS FOR FURTHER RESEARCH 5)... <\scneeetemenens neue 58 
APPENDIX A SCOUT HELICOPTER WORKLOAD SURVEY ..... An aa 6: Se, 
APPENDIX B ATTACK HELICOPTER WORKLOAD SURVEY ............ 71 


APPENDIX C SPEARMAN RANK CORRELATION SIGNIFICANCE LEVELS 83 


LIST OF REFERENCES oe @ eee ecmUmUOMmUlUOOmUCUC OU OCMC OCUCUC OCUCOrCUCUCU OrhUCUCcCOrmhUCUcCOrhUCUcCOrhUCUCcCOrmhUCUhOrmhUCUcCOrmhUCUcCOrhUCUcCOrmhUCUc OhUCUCOrmhUCOhlUhHhlhlUh 


ENITIAL DISTRIBUTION@EI ST ree eee eee 


vi 


Figure 1. 
Figure 2. 


Figure 3. 


Figure 4. 


Figure 5. 


Figure 6. 


Figure 7. 


Figure 8. 


Figure 9. 


Figure 10. 


LIST OF FIGURES 


MMos Wotkcitatmkoh CONCEDt .Mersl watt oo 6 5 10: ots aac Sra siene a-< 3 
Data minal. Voi SS CMe Mews. ccs eects ce te ciate tote. wo, eleepener eat 19 
Multiple Box Plots of Grouped Aviator Total 

po Meodeep tonite WeTvear ie prct scl co taceteeetawete sere eee aWae: Gia. CRC MNS, @: chou eaeeaes PA 
Plot of Attack Helicopter Pilot Workload 
Measurements Based on Individual-Task 
Pane Gi GOMpat 1S ONS ce steeicrck sieves «ais Go eucmemie te ee ceie whi ei 26 
Plot of Scout Helicopter Pilot Workload 
Measurements Based on Combined-Task Paired 
GCOMm ala SOMS meyers: ke he aceeeae oie oy eens ae c Seee we teem 26 
Plot of Aviator Workload Measurements Based 

on Individual Task Ratings by All Study 
Pale 1 alts  teecuete sat erecer cuore © ae. ois, cwecetans: 6 Seems -<ceneee eee 27 
Friedman Test Average Ranks of Aviator Groups 

for the Individual Task Paired-Comparisons .... 29 
Spearman Rank Correlations for the Individual- 
ask Pal red. Commas 1 SOMS: wre cisee occ 5 eee aaet sae Riess Roe aoe a2 
Highly Correlated Scout Helicopter Model (SCT) 
and Survey (SH-AB) Workload Values ............ 50 
Attack Helicopter Model (ATK) and Survey 


(AME) sWOrK VO acd Vales terersne seers, cee eens ace kos sees 50 


Va) 


Figure 


Figure 


Figure 


Figure 


Figure 


Figure 


Figure 


ee 


12. 


13. 


14. 


134 


16. 


LL 


Attack Helicopter Model (BSCATK) and Survey 

(AHR) Work loaGg Vy adwes <1. ie teteie a6 655.0 wien Sate! /e oS cee 51 
Scout Helicopter Model (SCT) and Survey 

(SH-CV) Workload Values «22.2252. ce ns ce ooo ee aul 
Scout Helicopter Model (BSCSCT) and Survey 
(SH-CV) Workload ValleS (2222. see ee eee SZ 
Attack Helicopter Model (ATK) and Survey 

(AH-AB) Workload Values ........... Riera. (et iets; 0a: See 52 
Attack Helicopter Model (BSCATK) and Survey 
(AH-AB) Workload Value€S 2... 0.622008 00 0 0 6 sie oe 
Scout Helicopter Model (SCT) and Survey (SHR) 
Workload ValuGs «2.5%... «ss. So ee Cf 53 
Scout Helicopter ModeL (BSCSCT) and Survey 


(SHR) Workload ValWes irre eee. ew os ee 54 


Vill 


TABLE 


TABLE 


TABLE 


TABLE 


TABLE 


TABLE 


TABLE 


TABLE 


TABLE 


TABLE 


TABLE 


TABLE 


TABLE 


TABLE 


10. 


els 


12. 


13. 


14. 


LIST OF TABLES 


ATTACK HELICOPTER FLIGHT TASKS 


SCOUT HELICOPTER FLIGHT TASKS 
SURVEY RESPONDENTS 3 ss srs imeiee. Sn eee ee ew 6s 0010 & ow 
INITIAL AVERAGE TASK RATING WORKLOAD VALUES 
INITIAL AVERAGE PAIRED-COMPARISON WORKLOAD 


VALUES 


WILCOXON SIGNED RANK TEST RESULTS FOR THE 
TASK RATINGS OF CORRELATED GROUPS 
WILCOXON SIGNED RANK TEST RESULTS FOR THE 
PAIRED COMPARISONS OF CORRELATED GROUPS ....... 
POOLED AVERAGE TASK RATING WORKLOAD VALUES 
POOLED AVERAGE PAIRED-COMPARISON WORKLOAD 
eee li) Bit oeeice oh airetin ice oa iatger es. are faye 6 are coy sume ts to qe teelictoiieieh eiveda oe aa sates 
RANK CORRELATION SIGNIFICANCE LEVELS FOR 
POOLED WORKLOAD VALUES 
RANK CORRELATION SIGNIFICANCE LEVELS OF MODEL 
VERSUS STUDY WORKLOAD VALUES, FOR ATTACK 


HELICOPTER INDIVIDUAL TASKS 


1X 


TABLE 15. RANK CORRELATION SIGNIFICANCE LEVELS OF MODEL 
VERSUS STUDY WORKLOAD VALUES, FOR SCOUT 
HELICOPTER INDDPVTDUAL VASK ares) << <6) ol cients cs > nae 45 
TABLE 16. RANK CORRELATION SIGNIFICANCE LEVELS OF MODEL 
VERSUS STUDY WORKLOAD VALUES, FOR ATTACK 
HELICOPTER COMBINED TASKS ...... oe see 0 6 0 oe © 6 eee 46 
TABLE 17. RANK CORRELATION SIGNIFICANCE LEVELS OF MODEL 
VERSUS STUDY WORKLOAD VALUES, FOR SCOUT 
HELICOPTER COMBINED LASKS Gaia. ee oe ccces 46 
TABLE 18. RANK CORRELATION SIGNIFICANCE LEVELS OF TLM 
INDIVIDUAL TASK WORKLOAD VALUES ........... oe 47 
TABLE 19. RANK CORRELATION SIGNIFICANCE LEVELS OF TLM 


COMBINED TASK WORKIOAD VALUES... ... 2. « +. eine ~ 47 


A°I 


AH-AB 


AH-CV 


EO-AB 


nO=CV 
CS 
MIDAS 
NOE 
NASA 
Psi 


SH-AB 


oHeeV 
TLM 
TLX 


TSU 


ABBREVIATIONS AND ACRONYMS 


Army-NASA Aircrew/Aircraft Integration Program 


Attack Helicopter Pilots of the Attack Helicopter 
Battalion 


Attack Helicopter Pilots of the Cavalry Squadron 


Enlisted Aerial Observers of the Attack Helicopter 
Battalion 


Enlisted Aerial Observers of the Cavalry Squadron 
Inter Communication System 

Man-Machine Integration Design and Analysis System 
Nap of the Earth 

National Aeronautics and Space Administration 
Pilot Steering Indicator 


Scout Helicopter Pilots of the Attack Helicopter 
Battalion 


Scout Helicopter Pilots of the Cavalry Squadron 
Task Loading Model 
Task Load Index 


Telescopic Sight Unit 





I. INTRODUCTION 


A. BACKGROUND 
1. Operator Error and the Design Process 

Operator error is the primary cause of many aircraft 
mishaps. "Even if some divine power could guarantee that all 
mechanical and electrical systems used in aviation were 100% 
reliable, there would still be accidents due to human error" 
(Kantowitz, 1988, p.158). This is especially true in the 
demanding environments associated with military aviation. As 
technological advances raise the performance levels of modern 
military aircraft, the mission employment expectations placed 
upon the aircraft and their operators also rise. Asa result, 
pilots encounter more difficult operational environments, 
experience higher workload demands, and frequently surpass 
their human performance capabilities. 

There is a concerted effort among design engineers to 
reduce operator workload through cockpit automation. However, 
a large gap exists between the desired effects of automation 
and its actual ramifications. Instead of lowered operator 
workload, many well intentioned designs result in prolonged 
training, illogical tasking, and excessive demands on pilot 
adaptability. (Army-NASA, 1990, p.2) 
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2.  Army-NASA Design Tool: MIDAS 
The U.S. Army and the National Aeronautics and Space 
Administration (NASA) are working together to create a human 
performance model that overcomes the problem of good 
intentions and poor results. The Army-NASA Aircrew/Aircraft 
Integration Program (A°l) is developing a computer simulation 
of aircraft crewstations and their operators. The simulation 
is to serve as a tool that can help ensure that human factors 
considerations are included in system design. 
The Man-Machine Integration Design and Analysis System 
(MIDAS) is one part of the A°I program's efforts. The MIDAS 
workstation presently simulates an AH-64 Apache helicopter, 
its pilot, and the environment in which they operate. MIDAS 
includes a variety of simulation submodels in areas such as 
anthropometry, vision, cockpit display layout, the pilot's 
cognition, and task loading. The synchronization of the MIDAS 
submodels results in a very capable design tool. (Army-NASA, 
1990) 
MIDAS developers envision an interactive computational 
system such as that illustrated in Figure 1. Use of such a 
system will improve the current design process in several 
ways: 
1. It enhances the engineer's efficiency and creativity. He 
or she can configure a cockpit, test it with a human 


model, and immediately evaluate the design's suitability 
in a variety of environments. 


2. Human factors considerations, often ignored during the 
initial phases of design, are a significant aspect of the 
feedback produced by MIDAS. 


3. Extensive use of computer modeling will reduce the 
expensive development of faulty prototypes, thus reducing 
the research and development costs associated with a 
procurement program. 


4. Although now configured as an Apache helicopter, MIDAS 
will be applicable to any man-machine vehicular system. 
(Army-NASA, 1990, pp.1-6) 


ARMY-NASA AIRCREW/ AIRCRAFT INTEGRATION 
PROGRAM (AD 


wur IS THE RAW GRAPHIC 
DATA/INFORMATION? 


WHAT ARE THE DETAILS 
TASK/COST/LOAD/CONFLCTS? 





Figure 1. MIDAS Workstation Concept (Army-NASA, 1990, p.8) 
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3. MIDAS Task Loading Model 

The measurement of pilot workload is an important 
aspect of the aircraft design process. "When workload is 
excessive, errors arise from the inability of the human to 
cope with high information rates imposed by the environment. 
When workload is too low, the human is bored and may not 
attend properly to the task at hand, also leading to error" 
(Kantowitz, 1988, p.159). 

The Task Loading Model (TLM) is one high resolution 
facet of MIDAS. The TLM manipulates state variables related 
to the operator and the aircraft to calculate the operator's 
relative workload. The TLM output classifies the pilot's 
individual flight tasks and task demands across the visual, 
auditory, cognitive, and motor dimensions. The classifi- 
cations and calculations generate workload data that designers 
may use to determine the level of human performance required 
for a specific system configuration. (Staveland, 1990) 

Four dimensions of human performance are measured by 
the TLM. These four dimensions represent the visual, 
auditory, cognitive and motor activities of a pilot. The TLM 
subdivides the four dimensions into several elements, 
including near and far for a visual activity, salient and 
masked for an auditory activity, planned and unplanned for a 
cognitive activity, and gross and fine for a motor activity. 
Each task is classified according to the four dimensions and 


4. 


the elements within the dimensions. The TLM then generates 


workload values along each dimension. (Staveland, 1990) 


B. TLM PROBLEMS 

Several problems and uncertainties currently are 
associated with the TLM. First, the nature of human workload 
is not fully understood. Although the model incorporates 
"Current research in multiple resource theory, scaling, 
workload, and perception" (Army-NASA, 1990, p.14), limitations 
in the field of behavioral psychology still limit the model. 
Second, current workload measurement mechanisms cannot capture 
all aspects of task loading and human performance. Thus, 
behavioral scientists skeptically view attempts to compile 
workload data. Third, the TLM output requires verification 
with credible workload data to insure that it reflects 
reality. However, even "suspect" helicopter pilot workload 
data suitable for comparison with the TLM output is not 
available. 

There is a critical need to verify the TLM output of 
workload values by comparing them with values obtained by 
other means. This is true despite the present lack of 
standard workload measurement techniques. "The workload 
problems of today will not wait until scientists develop 


perfect models. Fortunately, even incomplete models are 


useful in developing practical specifications" (Kantowitz, 
1988 soe bs 2 ee 
C. PURPOSE AND SCOPE 

The purpose of this study is to provide data that can be 
correlated with output from the A*I MIDAS TLM. The study 
consisted of three primary procedures. First, subjective 
workload data was collected from subject matter experts 
(helicopter pilots) by means of a survey form. Next, the 
survey data was analyzed to develop a robust, valid set of 
workload values. Finally, TLM output of workload values was 
compared with survey workload data to determine how closely 
the two data sets correlated. 

Chapter II of this thesis discusses the methodology used to 
collect subjective workload estimates from helicopter pilots. 
Chapters III and IV provide the data analysis techniques and 


results. Chapter V outlines conclusions and recommendations. 


II. METHODOLOGY 


A. STUDY OVERVIEW 

Verifying the reliability of the TLM output was a six-step 
process. The data used for the study came from a survey 
administered to US Army helicopter pilots in a classroom 
environment. This method of data collection allowed for input 
from 71 subject-matter experts in a short time frame and at 
minimal expense. The six steps of the process were: 


im idencitreation of helicopter flight tasks for which to 
determine workload values. 


2. Construction of survey forms which were used to collect 
pilot opinions about the workload levels related to these 
tasks. 

3. Determination of data analysis procedures. 

4. Administration of the surveys to Army helicopter pilots. 


5. Analysis of the workload data collected via the survey. 


6. Comparison of the survey data to the TLM output of 
workload values related to the same flight tasks. 


The remainder of this chapter further explains the first four 
steps of the above procedure. 
B. SCENARIOS AND TASKS 
1. Flight Scenario Development 
Flight tasks for which workload values were collected 
were identified through an analysis of flight missions 
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commonly carried out by the pilots who were surveyed. 
Similar, but not identical, flight scenarios were developed 
both for attack and for scout helicopter pilots, since their 
missions are closely related but vary in significant ways. 
The scenario settings used for the survey are provided in the 
survey forms, examples of which are included as Appendix A 
(scout helicopter) and Appendix B (attack helicopter). 

Each of the two scenario descriptions was developed by 
a pilot with experience in both the mission and the type of 
aircraft "flown" in the scenario. Each scenario represents 
approximately a two-minute slice of time froma typical attack 
or reconnaissance mission. Tasks occurring during the 
scenario are based on what a realistic cockpit crew is 
expected to do, under the assumption that the survey 
respondent is manipulating the flight controls. This creates 
a Single-pilot situation from which consistent task workload 
measurements could be drawn. 

2. Task Selection 

The specific tasks associated with each scenario were 
next identified. Of the more than 20 maneuvers conducted by 
the pilot throughout the scenario, nine items were selected as 
"individual tasks," which are defined as discrete activities 
carried out by a pilot while flying (see Tables 1 and 2). An 
activity is a physical or mental process associated with a 
helicopter flight mission. Activities may be described by the 
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four dimensions of human performance associated with the TLM: 
the visual, auditory, cognitive and motor dimensions. 

Additionally, the nine individual tasks were merged 
into six "combined tasks," also listed in Tables 1 and 2. 
Combined tasks consist of discrete events that overlap along 
a mission time line. For example, Tell crewmember unmasking 
is one individual task and Unmask aircraft is another individ- 
ual task. However, Tell crewmember "Unmasking" while unmask- 
ing the aircraft is a combined task. 

Since workload levels collected via the surveys were 
to be compared with those generated by the MIDAS TLM, it was 
important that the tasks included in the survey forms consist 
of a potentially wide variety of workload levels. The nine 
individual tasks selected from the two scenarios met this 
requirement. As a whole, they encompassed the range of 
loading evaluated by the TLM. They were selected intentional- 
ly to verify all dimensions of the model, as described in 
Chapter I. 

The nine individual tasks selected for the attack and 
scout helicopter pilots were exactly the same or parallel. 
For example, Task 4 for both groups of pilots was Hover. 
However, Task 8 for the attack helicopter pilots was Switch 
master arm to "arm," while for the scout pilots it was Turn up 
volume on ICS control panel. Both tasks involve reaching with 


the left hand and adjusting a switch or knob with the fingers. 


TABLE 1. ATTACK HELICOPTER FLIGHT TASKS 








TASK TASK 
: CODE INDIVIDUAL TASKS 


| 4 | Foi | Follow scout 

eae Se NOE deceleration 

3 | ext | Check torque meter and announce reading 
(Sat Tell crewmember "Unmasking" 
|| some 

re 


No 

3 

5 
Unmask aircraft 

- 

al 

2 


Lstn | Listen to radio transmission from scout 
(direction to fire, range, target) 
Switch master arm to "arm" 


Intp | Interpret gunner's position of TSU based on 
PSI information 





COMBINED TASKS 


fh 2 Weaees | NOE deceleration behind scout 
Ccktq | Check torque meter and announce reading 
while NOE decelerating 
Cckhv | Check torque meter and announce reading 
while hovering 
Cumsk | Tell crewmember "Unmasking" while unmasking 
aircraft 


S| 

5 Clstn | Listen to radio transmission from scout 
(direction to fire, range, target) while 

6 







switching master arm to "arm" 









Cintp | Interpret gunner's position of TSU, based 
on PSI information, while receiving target 


handoff from scout 
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TABLE 2. SCOUT HELICOPTER FLIGHT TASKS 


TASK TASK 
No. CODE INDIVIDUAL TASKS 


| | Fotw | Follow another scout 
Ha | wove |tover 
Tell crewchief "Unmasking" 

| 6 | umsk | unmask aircratt 
| | turn | urn up volume on Ics control panel 


Detm | Determine present location (6 digit grid) 
as crewchief holds map 


COMBINED TASKS 


Pls | cacar NOE deceleration behind another scout 


2 Ccktgq | Check torque meter and announce reading 
while NOE decelerating 


| Cckhv | Check torque meter and announce reading 
while hovering 


aircraft 


5 Clstn | Listen to crewchief describe what he sees 
while you turn up volume on ICS control 
panel and hover 

6 


Cdetm | Determine present location (6 digit grid) 
as crewchief holds map and you listen to 
him describe what he sees 


i Cumsk | Tell crewchief "Unmasking" while unmasking 





aa 


C. SUBJECTIVE WORKLOAD MEASUREMENT TECHNIQUES 
1. Selection of Measurement Techniques 
Several means of eliciting workload judgements fron 
pilots were considered. The desired characteristics of a 
workload measuring mechanism were as follows: 
1. Reliable and credible results are obtained. 


2. An appropriate level of effort is required of the 
respondent. 


3. The resulting subjective measurements can be quantified. 
4. Administration to groups of 15 or more is easy. 
5. Data analysis is straightforward. 

Two established techniques were chosen for this study, 
since both met the above requirements. They were subjective 
workload rating scales and the psychometric technique known as 
Thurstone's Method of Paired Comparisons (ARI Report 851, 
1989, pp.81-122). Examples of both of these techniques are 
included in the example survey forms located in Appendix A and 
Appendix B. 

2. Rating Scales 

Workload rating scales require the respondents to 
assign a numerical value (from 0 to 10 for this study) to each 
task, thereby indicating its level of difficulty. The use of 
rating scales was advocated by the designer of the MIDAS TLM, 
resulting from his experience with the NASA-Task Load Index 


(TLX) (Hart & Staveland, 1987). The TLX is a highly refined 
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subjective workload assessment scaling technique based on 
rating scale methodologies. Rating scales meet the five 
requirements listed above for a workload measuring mechanism. 
They allow for any number of subjects, tolerate missing 
responses, and facilitate efficient statistical analysis. A 
potential drawback is that raters become frustrated when 
required to make large numbers of judgements. Thus, reducing 
possible frustration was a major reason for including only 
nine tasks on the survey form (Zatkin, 1983). 
3. Thurstone's Method 

Thurstone's method utilizes several judges who compare 
each item ina set against each of the other items, one at a 
time. This method results in an ordering of the items 
according to a relative value of magnitude interdependent on 
every other compared item. For the workload measurements, 
each pilot determined which of two compared tasks was more 
aifficult. Judgements were made for each set of two until all 
individual or combined tasks had been compared with all the 
other individual or combined tasks. From these comparisons a 
scale of relative task difficulty emerged. (Thurstone, 1963) 

A drawback to Thurstone's Method is the potentially 
cumbersome number of pairwise comparisons required of the 
respondent. To compare n items against each other requires 
n(n-1)/2 judgements. Workload measurements for this study 
required 36 comparisons of the nine individual tasks and 15 
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comparisons of the six combined tasks. This number of 
pairwise comparisons and the accompanying scale ratings 
provided a challenging, but not overwhelming endeavor for the 
pilots participating in the survey. 
D. SURVEY CONSTRUCTION 

1. Survey format 

Two separate survey forms were used to measure 
helicopter pilot experience with workload during flight, one 
for scout helicopter pilots and the other for attack helicop- 
ter pilots (see Appendices A and B). Each was designed to be 
completed in approximately 15-20 minutes. The 12-page survey 
forms included an introduction, instructions on how the survey 
should be filled out, and the scenario setting. The survey 
itself was divided into four sections: 

1. Individual task subjective rating scales. 
2. Combined task subjective rating scales. 
3. Individual task pairwise comparisons. 
4. Combined task pairwise comparisons. 
Pilot background information also was collected. 

Ratings and comparisons were randomized within each 
section for each individual survey form. The four sections 
also were shuffled into their 24 different permutations, to 
create 24 unique formats. Rosenthal points out that "very 
large questionnaires can tend to discourage the subjects from 
answering. A way around this problem is to vary the format in 
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order to keep the subjects interested" (Rosenthal, 1984, 
p.-134). Besides maintaining subject interest, randomization 
also reduced the chances that judges might respond according 
to a set contextual flow of the survey material (Rosenthal, 
1984). 
2. Survey Content 

The first page of each survey form provided background 

information to the pilots. Several points were highlighted: 


1. The survey's purpose: attain workload information to aid 
in the improvement of future Army helicopter cockpits. 


2. Emphasis on candid responses, with no right or wrong 
answers. 


3. Individual anonymity and the pooling of response data. 


4. The uniqueness of each survey form due to question 
randomizing. 


These points were discussed to help control what Rosenthal 
calls "demand characteristic: the subject's perception of his 
or her role and of the experimenter's hypothesis." Clarifying 
the pilots' role made them more sensitive to the aims of the 
survey, thereby motivating a response, while instilling 
confidence in their ability to respond appropriately. 
(Rosenthal, 1984, p.105) 

The second page of the form consisted of detailed 
instructions indicating how to respond. Specific, relevant 
examples were included. This was followed by a page contain- 


ing a one-paragraph scenario description providing the context 
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within which the pilot was to respond. The setting was that 
of a typical tactical flight mission. Applicable parameters 
such as crew mix, weather conditions, and the enemy situation 
were described. The forms for the actual ratings and compari- 
sons followed. The final page contained a short questionnaire 
regarding the aviation experiences and qualifications of the 
responding pilot. 

The two survey forms were reviewed for accuracy of 
content and wording by two Army instructor pilots. They 
provided helpful feedback by recommending current aviation 
phraseology, such as changing "copilot" to "crewmember." They 
verified the sound structure of the scenarios and reported 
that the survey was appropriate for the objectives that were 
to be met. 

3. Minimizing the Social-Desirability Response 

A concept from the field of behavioral science that 
influenced survey construction was what is referred to as the 
"social-desirability response set" (Rosenthal, 1984, pp.135- 
139). With respect to helicopter pilots, this mindset would 
tend to cause them to tailor their responses so they would 
appear to be competent pilots who rarely experience demanding 
workload. Three techniques were utilized to minimize the "I 
can handle anything" mentality common among pilots. First, 
anonymity was emphasized so that no individual could be linked 
to a specific response. Second, the forced choices associated 
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with Thurstone's paired comparisons reguired the respondent to 
specify that one task was harder than another. This prevented 
respondents from claiming that none were difficult. The third 
technique involved the interaction of the survey administrator 
with the pilots who were completing the survey, as discussed 
in the next section. 
E. SURVEY ADMINISTRATION 
1. Survey Respondents 

Surveys were administered to pilots assigned to the 
Seventh Infantry Division (Light) located at Fort Ord, 
California. Pilots from the 2/9 Cavalry Squadron completed 
the survey on 29 March 1991. Pilots from the 1/123 Attack 
Helicopter Battalion completed the survey on 23 May 1991. 
Table 3 shows the number of respondents by category. 

A third group of respondents consisted of Enlisted 
Aerial Observers from the same Squadron and Battalion, 
surveyed on the same dates. Aerial Observers are enlisted 
soldiers who accompany scout pilots during a mission. 
Although their tasks do not usually include hands-on flying, 
each has at least 60 hours of flight training prior to 
reporting to an aviation unit. As flight crewmembers working 
in the cockpit, they have experience with the surveyed tasks. 
Thus, their responses contributed a third pool of information 


to the data set. 
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TABLE 3. SURVEY RESPONDENTS 


2/9 Cavalry 1/123 Total by 
Sqdn Attack Bn Group 
Attack Helicopter 
Pilots 


Scout Helicopter 

Pilots 

Aerial Observers ees 
Total by Unit 





2. Survey Administration Procedures 
The surveys were administered in a classroom environ- 
ment during regularly scheduled pilot meetings. Before 
receiving the questionnaire, the pilots and observers were 
given a short briefing by the survey administrator, an Army 
captain. He used an overhead projector to provide necessary 
information to the subjects and emphasized the following: 


1. The possible effect of their efforts on future aircraft 
design. 


2. An overview of the A*I and MIDAS systems. 


3. A short discussion of human factors and its role in 
aircraft design. 


4. Examples of survey questions and responses. 
At the end of the briefing, the survey forms were distributed 
to the respondents by each unit's instructor pilots. Plat 


forms were completed within 15-25 minutes after distribution. 
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III. ANALYSIS OF STUDY DATA 


A. DATA ANALYSIS SCHEME 


The data collected from the 71 subjects surveyed was 


analyzed according to the scheme depicted in Figure 2. The 
Pilot 7 Data Box Ptots 
Transform Raw Data 
Task Ratings Pairwise Comparisons 
Individual Combined Individual Combined 
Scatter Plots 


Nonparametric Tests 
(Friedman, iSpouinGn Wilcoxon) 


Conelation of Remaining Survey Data 


Comparison of Survey Data With Mode! Data 
Figure 2. Data Analysis Scheme 


concepts guiding the analysis were: 


1. Determine the degree of homogeneity among~ the 
respondents. 
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2. Attain workload values for the tasks evaluated by the six 
groups of subjects from a Cavalry Squadron and an Attack 


Helicopter Battalion, each consisting of attack 
helicopter pilots, scout helicopter pilots, and aerial 
observers. 


3. Filter the groups of data by means of nonparametric 
statistical tests. 


4. Accept a final pool of workload measurements that meet 
the above criteria. 


These concepts were utilized to provide an internally 
consistent set of study data. 
B. PILOT BACKGROUND INFORMATION 

The last page of the workload survey consisted of 
questions intended to establish each subject's aviation 
background and experience (see Appendix A or B). The pilot 
background data was graphically analyzed using Box plots 
(Rice, 1988, pp.336-337) in order to compare the aviators in 
the 2/9 Cavalry Squadron with those of the 1/123 Attack 
Helicopter Battalion. Three particular measures were 
analyzed: 

1. Total flight time. 


2. Flight time in the aviator's primary aircraft: either the 
attack helicopter or the observation helicopter. 


3. Flight time as a pilot-in-command. 


Figure 3 shows a multiple Box plot of the total flight 
time of all the subjects. The first plot represents the 
attack helicopter pilots of the Attack Battalion (AH-AB) 
alongside those of the Cavalry Squadron (AH-CV) in the second 
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plot. 


The third and Lourth plots (SH=-AB<and SH-CV) are those 


of the Scout pilots. The two groups of observers (EO-AB and 


EO-CV) are plotted last. 


3588 


26e5 


1758 


TOTAL FLIGHT TIME 


67s 





AH-AB AH-CU SH-AB SH-CU EO-AB EO-CY 


Figure 3. Multiple Box Plots of Grouped Aviator 
Total Flight Time 


These Box plots provide an efficient method of analysis 


for several groups of data. Several observations can be made, 


based on the complete set of Box plots analyzed during this 


study. These observations include the following: 


lg 


When grouped within their respective units, the attack 
helicopter pilots of the Cavalry Squadron and Attack 
Battalion have very similar backgrounds in total flight 
time, attack helicopter flight time, and pilot-in-command 
flight time. 


There was one attack helicopter pilot outlier, a pilot in 
the Cavalry Squadron with almost 3000 hours of total 
flight time. 
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3. The grouped scout helicopter pilot data indicated that 
subjects from the Cavalry Squadron averaged approximately 
100 hours more experience in the three flight time 
categories than those from the Attack Battalion. 


4. Each scout helicopter unit had one outlier with slightly 
more than 2000 hours total flight time. 


5. Only total flight time data was analyzed for the enlisted 
aerial observers. Box plots for observers from both 
units are almost identical. 

C. DATA TRANSFORMATION AND SUMMARIZATION 

The data collected on the survey forms was transformed 
into 24 response vectors of either six or nine values. Each 
vector was derived from grouped subject responses for either 
the individual or combined task data from the rating scales or 
paired-comparisons. In other words, these 24 vectors 
represented responses from the six aviator groups for the four 
sections of the survey. 

The rating scale responses were transferred from each 
Survey form to a spreadsheet. There they were recorded, 
summed, and averaged. The resultant values are listed in 
Table 4, where smaller values denote lower workload levels. 
The paired comparison responses were transferred from the 
survey forms into pooled groups of workload: values by means of 
Thurstone's Method (Thurstone, 1963, pp. 67-81). The 
particular method used, Thurstone's Case V (Dunn-Rankin, 1983, 


pp. 79-82), resulted in the values listed in Table 5. 
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TABLE 4. INITIAL AVERAGE TASK RATING WORKLOAD VALUES 


INDIVIDUAL TASKS 
|_| attack pizots | scout Pilots 


Code 
ee ee ee a 
No. Bn Sqdn Bn sqdan Bn Sqdn 
| a | 3.87 | 4.19 | 2-17 | 2-08 | 2-83 | 4-29 
a ees re ee 
2.56 2.81 


poe | 2.06 | 2.58 | 2.39 | 2.00 | 2.33 | 1.56_ 
po | 3.42 | 2.92 | 5.50 | 5.92 | 4-50] 6.56_ 


COMBINED TASKS 


pa | 2.33 | 4.05 | 4.29 | 3.50 | 4.09 | 4.69 
pa | oso | a.62 | 220 | 2.39 | 2.67 | 1.56 
ps | 3.27 | 4.00 | aa3 | 3.08 | 4.33 | 3.29 
pos | sso | 435 | soe | 5.58 | 5.83 | 5a 


a a ee ee 
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TABLE 5. INITIAL AVERAGE PAIRED-COMPARISON WORKLOAD 
VALUES 


INDIVIDUAL TASKS 
|_| attack pirots | scout Pilots 
Group 
Code 


Task Attack Cav Attack Cav Attack Cav 
No. Bn Sqdn Bn Sqdn Bn Sqdn 



















2 

3 e 

s_| 0.00 | 0.00 | 0.00 | 0.00 | 0.00 
_ aso | 2.76 | 2.28 | 1.19 | 1-88 | 0.46 

7 


COMBINED TASKS 


| 
| 0.53 | 0.77 | 1.00 | 0.30 | 0.78 | 0.00 










1 
2 
3 


: 
ee ee ee eee ee 


a ee ee ee 
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The workload values listed in Tables 4 and 5 are 
characterized as follows: 

1. Group codes, identifying columns of workload values, 
correspond with the similarly-labelled graphic data for 
the Squadron and Battalion used throughout this study 
(See Figure 3). 


2. Tasks are listed in order of their occurrence during the 
flight scenario. 


3. The greater the workload value, the more difficult the 
task. 


4. Thurstone's method assigns the easiest task the value 
zero. Higher-valued tasks are assigned a quantitative 
workload measure according to their relative difficulty 
in relation to the other tasks. 

5. The bottom row in each table indicates the number of 
respondents, n, whose responses were used to calculate 
each column's workload values. 

These 24 vectors of workload data provide the input for the 
subsequent analysis and tests to follow. 
D. PLOTS OF WORKLOAD DISTRIBUTION 

Plots of the workload data in Tables 4 and 5 provided a 
quick graphic assessment of the distribution of workload 
according to task (Chambers, 1983, pp. 82-86). Most of the 
plots constructed for the study compared the responses of one 
group of aviators from the Cavalry Squadron with those in the 
same type of group from the Attack Helicopter Battalion. 
Three representative graphs are depicted in Figures 4, 5, and 
6. 

Figure 4 shows the plot of individual-task paired- 


comparison workload values for the attack helicopter pilots in 


Zo 


both the Cavalry Squadron and the Attack Helicopter Battalion. 
Figure 5 shows combined-task paired-comparison workload data 
for the scout pilots of both units. Figure 6 shows a plot of 


all of the individual task ratings provided by all subjects, 


WORKLOAD 
ws 





4 6 
INDIVIDUAL TASKS 


Figure 4. Plot of Attack Helicopter Pilot Workload 
Measurements Based on Individual-Task Paired 
Comparisons 


WORKLOAD 





2 3 
COMBINED TASKS 


Figure 5. Plot of Scout Helicopter Pilot Workload 
Measurements Based on Combined-Task Paired 
Comparisons 
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averaged according to their six respective aviator and unit 
groups. All three plots indicate that there is a general 


trend toward like responses among the groups of aviators. 


WORKLOAD 





INDIVIDUAL TASKS 


Figure 6. Plot of Aviator Workload Measurements Based on 
Individual Task Ratings by All 8tudy 
Participants 


E. NONPARAMETRIC TESTS 

Several nonparametric statistical tests of the data were 
employed to investigate possible trends and to determine if 
responses were consistent among the subjects. These tests 
were used to explore the validity of combining the workload 
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data vectors into fewer pools of data, with more respondents 
Contributing “to @ach pools 
1. Friedman Test 

The Friedman test for randomized block designs 
(Mendenhall, 1990, pp. 702-704) was used to determine if the 
responses of each pilot grouping came from identical 
distributions. The significance level for this test was set 
ata. oO. 05: Thus, if the groups tested attained a p value 
greater than 0.05, then they would be considered to come from 
the same underlying distribution. 

Friedman tests were conducted on data for each of the 
four sections of the survey. Individual- and combined-task 
ratings and paired comparisons for each of the six aviator 
groups were tested as randomized blocks. Figure 7 graphically 
portrays the Friedman Test results for one section of the 
survey, the individual-task paired comparisons. The varying 
bar lengths indicate that the six groupings appear to differ 
in the average of the ranks given to workload values. This is 
numerically supported by the p value of 0.0001 obtained from 
the Friedman test for this randomized block. 

Similar results were found for initial average 
workload values from the other three survey sections. Table 
6 shows the p values for data from all four sections. Thus, 
for each of the four sections, it cannot be assumed that all 
Six groups of respondents are from the same underlying 
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distribution. Therefore, all six response vectors should not 
be combined into one data pool for any of the four survey 


sections, as none of the p values exceeded 0.05. 











AVERAGE RANK | 


Figure 7. Friedman Test Average Ranks of Aviator Groups 
for the Individual Task Paired-Comparisons 


TABLE 6. FRIEDMAN TEST P VALUES 


SECTION P VALUE IMPLICATION 
Individual Task Ratings 0.0071 | Do not pool data from 
top half TABLE 4 
Combined Task Ratings O70114 4) De not) pool data trom 
bottom half TABLE 4 
Individual Task Paired 0.0001 | Do not pool data from 
Comparisons tep Nalf TABLE 5 
Combined Task Paired 0.0448 | Do not pool data from 
Comparisons bottom half TABLE 5 


29 














2. Spearman Rank Correlations 

Since the Friedman test indicated that the data could 
not be combined into one data pool, Spearman rank correlations 
(Hamburg, 1987, pp. 576-578) were utilized to determine if 
data from any of the four sections could be pooled for any of 
the six aviator groups. Comparisons were made between like 
aviators of the Attack Battalion and the Cavalry Squadron. 
For example, the responses of the attack pilots of the Attack 
Helicopter Battalion (AH-AB) were correlated with those of the 
Cavalry Squadron (AH-CV) attack pilots. The significance 
level of 0.006 (shown in the upper left portion of Table 7) 
indicates a high degree of correlation between these two 
groups of responses. 

Of the groups of aviators, those that correlated with 
a Significance level less than 0.05 were considered to be 
Candidates for data pooling. Table 7 shows the significance 
levels associated with the Spearman rank correlations of the 
Six aviator groups for the individual-task paired comparisons. 
Only the correlations of like groups (those performing similar 
missions) are included. Additionally, scout pilot and 
enlisted observer comparisons were only made when the two 
groups were in the same unit. For instance, the responses of 
the Cavalry Squadron Observers (EO-CV) were correlated only 


with Cavalry Squadron scout pilots (SH-CV), not with Attack 
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Battalion scouts (SH-AB). However, cross unit observer 
comparisons were made (EO-AB with EO-CV). 

The results of the Spearman rank correlations for 
individual-task paired comparisons are more favorable to data 
pooling than are results obtained for data from the other 
Survey sections (Significance levels for the Spearman rank 


correlations of the other three sections of the survey are 


located in Appendix C). The bold numbers indicate groups 
whose correlations are statistically significant (i.e., less 
maam/0.05). 


TABLE 7. RANK CORRELATION SIGNIFICANCE LEVELS 


INDIVIDUAL-TASK PAIRED COMPARISONS 
|| sana sH-aB | sH-cv | Eo-aB | Eo-cv 





Figure 8 is a graph of the Spearman rank correlations 
for the five comparisons of data from the individual-task 
paired comparisons where correlations are statistically 


acceptable for pooling (AH-CV and AH-AB, SH-CV and SH-AB, SH- 
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AB and EO-AB, SH-CV and EO-CV, and EO-AB and EO-CV). These 
five comparisons correspond with the five bold significance 
levels in Table 7. The Spearman test results’ (low 
Significance levels and high correlations) for these five 
group comparisons indicates similar underlying distributions 


for each pair and suggests that pooling may be appropriate. 


————————————————————— eee 

A ee SL SS 
AHABBAH-CV) | CS 

<<a 


SH-AB & SH-CV 


SH-CV & EO-CV 


0.0 0.2 0.4 0.6 0.8 WG 
CORRELATION 





Figure 8. Spearman Rank Correlations for the Individual- 
Task Paired Comparisons 


3. Wilcoxon Signed Rank Test 
Before accepting the groupings of data suggested by 


the Spearman test, a final nonparametric test was conducted. 
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The Wilcoxon signed rank test (Mendenhall, 1990, pp. 680-681) 
provided a method of determining the sameness or difference of 
the aviator workload response distributions. The same aviator 
group comparisons made during the Spearman test were made with 
the Wilcoxon test. The p values generated by the Wilcoxon 
test are listed in Table 8 for the task ratings and in Table 
9 for the paired comparisons. P values greater than a 
significance level of 0.05 imply that the underlying 
distributions are similar. Values in this category are printed 


in bold type. 


TABLE 8. WILCOXON SIGNED RANK TEST RESULTS FOR THE TASK 
RATINGS OF CORRELATED GROUPS 


Compared Groups No. of 
poles p value pairs p value 


Tables 9 and 10 show the following: 



















1. Workload values may be pooled for both the individual- 
and combined-task ratings for AH-AB & AH-CV, SH-AB & SH- 
CV, and SH-AB & EO-AB. 
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TABLE 9. WILCOXON SIGNED RANK TEST RESULTS FOR THE 
PAIRED COMPARISONS OF CORRELATED GROUPS 


Attack Bn/Cav Sqdn No. of No. of 

Samples Pairs p value pairs 
AH-AB & AH-CV 
SH-AB_& SH-CV 


SH-AB_& EO-AB | 0.08 
SH-CV_& BO-CV 


2. Workload values may be pooled for only the combined-task 
ratings for EO-AB & EO-CV and SH-CV & EO-CV. 


co-ap & Eo-cv_ || 9 | oon 





3. Workload values may be pooled for both the individual- 
and combined-task paired comparisons for AH-AB & AH-CV 
and SH=CV & EO-CyV. 

4. Workload values may be pooled for only the combined-task 
paired comparisons for SH-AB & SH-CV, EO-AB & EO-CV, and 
SH-AB & EO-AB. 

Results obtained from the Wilcoxon signed rank test indicate 
that other values cannot be pooled. The test showed that the 
underlying distributions were not the same. 
F. POOLED WORKLOAD RESPONSES 

Pooled workload values resulting from the filtering effect 
of the nonparametric tests are listed in Table 10 for the task 
ratings and Table 11 for the paired comparisons. The data in 
Tables 10 and 11 represents the highest level of pooling that 


can be justified by the nonparametric tests. In other words, 


data from the groups of aviators was pooled as shown in Tables 
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10 and 11 if the Spearman test and the Wilcoxon test signifi- 
cance level values are shown in bold type in both Table 7 (or 
Appendix C) and Table 8 or 9. 

The values listed in Tables 10 and 11 were generated by 
recomputing the raw data from the survey forms. The 
recomputations involved transforming the pooled aviator 
response data into appropriate task ratings and Thurstone 
paired-comparison values. This process was carried out only 
for data groups shown in Tables 4 and 5 which were highly 
correlated according to the Spearman test, and were determined 
to be of the same underlying distribution by the Wilcoxon 
test. Data that did not meet these requirements was not 
included in Tables 10 and 11. Thus, four columns in Table 10 
and four columns in Table 11 are blank. 

Absent from Tables 10 and 11 are pooled workload values 
for the enlisted aerial observers (EO-AH and EO-CV). As these 
aviators are not qualified pilots, their responses considered 
alone may not accurately portray workload. Therefore, the 
observers' responses were included in Tables 10 and 11 only 
when they agreed with those of the scout pilots, according to 
the Spearman and Wilcoxon tests. The pooled workload values 
of the two right hand columns of Tables 10 and 11 include 
responses from the enlisted aerial observers pooled with those 


er) the Scout pilots. 
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TABLE 10. 


INDIVIDUAL TASKS 


All 
Scout 
Pilots 


Cannot be 
pooled 
according 
to 
Spearman 
Test 


Attack Bn 
Scout & 
Obser 


Cannot be 
pooled 
according 
to 
Spearman 
Test 


POOLED AVERAGE TASK RATING WORKLOAD VALUES 


Cavalry 
Sqdn Scout 
& Obser 


Cannot be 
pooled 
according 
to 
Wilcoxon 
Test 


Cannot be 
pooled 
according 
to 
Spearman 
Test 
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TABLE 11. POOLED AVERAGE PAIRED-COMPARISON WORKLOAD 
VALUES 









INDIVIDUAL TASKS 
























All 
Attack 
Pilots 


All Attack Bn Cavalry 
Scout Scout & Sqdn Scout 
Pilots Obser & Obser 


Cannot be pooled 
according to 
Wilcoxon 
test 






COMBINED TASKS 


cannot be 





Cannot be 











a ee ae 
according 1.99 1.02 according 
eS - 
Spearman Spearman 






—e.00 [0.00 test 

Pp a72 | 0.94 
rae ae 

ee ea 
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G. CORRELATION OF RATINGS WITH PAIRED~COMPARISONS 

Spearman's rank correlations were used with the pooled 
data of Tables 10 and 11 to verify the internal consistency of 
aviator responses. Workload ratings for the individual and 
combined tasks were compared with the paired-comparison 
workload values for the same tasks for those cases where both 
Table 10 and Table 11 had data in the same columns. 

For the individual tasks the only correlation that met 
this requirement was that for the attack helicopter pilots. 
The significance level of 0.007 for the Spearman correlation 
of all attack helicopter pilot individual-task workload 
ratings (AHR, Table 10) with related paired-comparison 
workload values (AHP, Table 11) indicates consistent attack 
helicopter pilot responses throughout the survey for these 
tasks. 

Combined-task workload values from Tables 10 and 11 also 
were tested. Table 12 provides the significance levels found 
for the Spearman correlations. Only scout helicopter pilot 
and enlisted aerial observer response data are included, as 
the attack helicopter pilot responses could not be pooled. 
All values in Table 12 are below the 0.05 significance level, 
suggesting consistency in the scout and observer responses for 


the combined tasks. 
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Results of the nonparametric tests prevented pooling of 
scout helicopter pilot individual task responses for the 
Attack Helicopter Battalion and the Cavalry Squadron. Thus 
Spearman correlations were performed on the initial workload 
values for the individual tasks of the scout helicopter pilots 
(Tables 4 and 5). Attack Battalion scout pilots' individual- 
task ratings (SH-AB, Table 4) and paired-comparisons (SH-AB, 
Table 5) correlated with a Significance level of 0.013. 
Cavalry Squadron scout pilots' individual-task ratings (SH-CV, 
Table 4) and paired comparisons (SH-CV, Table 5) correlated 
with a significance level of 0.008. As with the pooled 
observations above, the significance levels for correlation 
between the two kinds of initial workload values indicates 
consistency among the aviators' responses. 


TABLE 12. RANK CORRELATION SIGNIFICANCE LEVELS FOR POOLED 
WORKLOAD VALUES 


| COMBINED TASKS: RATINGS AND PAIRED COMPARISONS | 
SOR-AB | SOR-CV 


0.036 | 0.028 | 0.035 | 


0.000 
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Nonparametric test results also prevented pooling of 
attack helicopter pilot combined-task responses for the Attack 
Helicopter Battalion and the Cavalry Squadron. Thus, as with 
the scout pilot comparisons above, Spearman correlations were 
performed on the initial workload values (Tables 4 and 5). 
Attack Battalion attack pilots' combined-task ratings (AH-AB, 
Table 4) and paired comparisons (AH-AB, Table 5) correlated 
with a significance level of 0.048. Cavalry Squadron attack 
pilots' combined-task ratings (AH-CV, Table 4) and paired 
comparisons (AH-CV, Table 5) correlated with a significance 
level of 0.243. From these correlations, the Attack Battalion 
attack pilots' combined-task workload values are consistent 
for the ratings and the paired comparisons. However, the high 
Significance level (0.243) for the Cavalry Squadron does not 
Support consistency between the responses to the ratings and 
the paired comparisons. 

For the correlations of the task rating workload values 
with the paired-comparison workload values, low significance 
levels indicate high correlations. The results discussed 
above show that the subjects' responses were consistent 
throughout the survey. Furthermore, test results confirm 
robustness of the data and a high degree of reliability among 
the selected groups of responses. Based on these tests, the 
data in Tables 10 and 11 and selected values of Tables 4 and 
5 (except for Cavalry Squadron attack helicopter pilot 
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responses) can be confidently used as baseline workload values 
for the tasks and conditions considered. These baseline, 
empirically-determined values serve as "ground truth" data 
that can be used to validate the Midas Task Loading Model's 


analytically-produced values for the same tasks. 
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IV. COMPARISON OF MIDAS MODEL DATA WITH STUDY DATA 


A. TASK LOADING MODEL WORKLOAD VALUES 

The MIDAS TLM generates workload data at ae higher 
resolution than was measured by the survey forms. The TLM 
measures workload along four dimensions of human performance. 
These dimensions are the visual, auditory, cognitive, and 
motor activities performed by an aviator while flying. In its 
present configuration, the TLM averages workload values 
generated along each of the four dimensions for each of the 
individual and combined tasks. The tasks modelled in the TLM 
are comparable to the nine individual tasks and six combined 
tasks used in the pilot survey. The result is a set of task 
workload values as listed in Table 13. The numbers represent 
relative workload on a scale of 0 to 100, with larger numbers 
indicating higher workload. (Staveland, 1990, pp.6-9) 

Two different techniques were used with the TLM system to 
create workload values for the scout and attack helicopter 
pilot tasks. The first technique involved measuring only the 
workload generating attributes of the human performance 
dimensions involved ina task beyond the activity of flying. 
That is, only workload related to human performance above the 


basic activity of flying was calculated. This technique 
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generated the first two columns of data in Table 13 (ATK and 
SCT). The second technique combined the workload attributes 


of a task with an underlying activity of basic flying. Thus 
TABLE 13. TLM-~GENERATED WORKLOAD VALUES 
INDIVIDUAL TASKS 


Group ATK SCT BSCATK BSCSCT 
Code 


Task 


= 
O 





the workload of each task included the supposition of an 
ongoing basic hands-on flying activity. This technique 
generated the last two columns of data in Table 13 (BSCATK and 
BSCSCT). (Staveland, 1990, pp.19-23) 

From a visual inspection of the data in Table 13, it 
appears that the TLM-generated values for attack and scout 
pilots are similar in columns ATK and SCT and in columns 
BSCATK and BSCSCT, respectively. Except for tasks 7 and 8, 
the TLM values indicate that the scout and attack helicopter 
pilots should experience almost identical levels of workload 
for a given task. An additional observation is that many 
tasks within a column of data have identical or near identical 
workload values. For example, the individual-task portion of 
column BSCSCT contains four workload measurements of 42, two 
of 46, and one each of 44 and 45. According to the TLM model 
results, eight out of nine tasks involve essentially the same 
amount of workload. 

B. RANK CORRELATION OF MODEL DATA WITH SURVEY DATA 

The TLM model-generated workload values were compared with 
the survey workload values in a procedure similar to that 
described in Chapter III, Section G, where the task ratings 
were correlated with the paired comparisons using the Spearman 


rank correlation technique. The survey workload values used 
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for this correlation are from Tables 4, 5, 10, and 11. The 
model data used is all from Table 13. 

The results of the Spearman rank correlation technique are 
provided in Tables 14 to 17. Each of the first four tables 
corresponds to one of the four sections of the survey. Tables 
18 and 19 show the results of correlating the TLM with itself. 
Significance levels below 0.05, printed in bold type, indicate 


high correlation. 


TABLE 14. RANK CORRELATION SIGNIFICANCE LEVELS OF MODEL 
VERSUS STUDY WORKLOAD VALUES, FOR ATTACK 
HELICOPTER INDIVIDUAL TASKS 





TABLE 15. RANK CORRELATION SIGNIFICANCE LEVELS OF MODEL 
VERSUS STUDY WORKLOAD VALUES, FOR SCOUT 
HELICOPTER INDIVIDUAL TASKS 


Survey Group Code 
SH-AB SHi=CV SH-AB SsH=CV 
(ratings) | (ratings) | (comparisons) } (comparisons) 


! 
0.027 | 0.521 0.049 0.225 
BSCSCT 0.628 0.504 0.852 0.604 
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The following observations may be made from the six tables: 


1. No significant correlations were found for the attack 
helicopter individual or combined tasks (Tables 14 and 
16) nor the scout helicopter combined tasks (Table 17). 


2. Two significant correlations were found for the scout 
helicopter individual tasks (Table 15). The TLM's scout 
workload values were strongly correlated with both the 
ratings and paired-comparison values from the Attack 


Battalion scout pilots' (SH-AB). However, only six scout 
pilots contributed to these two categories of survey 
data. 


TABLE 16. RANK CORRELATION SIGNIFICANCE LEVELS OF MODEL 
VERSUS STUDY WORKLOAD VALUES, FOR ATTACK 
HELICOPTER COMBINED TASKS 


Survey Group Code 
AH-AB AH-CV AH-AB AH-CV 
|} (ratings) | (ratings) | (Comparisons) (Comparisons 
| 


| 0.565 1.000 0.482 0.949 
BSCATK 1.000 0.841 0.844 1.000 














TABLE 17. RANK CORRELATION SIGNIFICANCE LEVELS OF MODEL 
VERSUS STUDY WORKLOAD VALUES, FOR SCOUT 
HELICOPTER COMBINED TASKS= 















Survey Group Code 


0.897 0.565 O.7655 0.565 
0.490 0.476 0.604 0.476 
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The significance levels for TLM-survey data correlations 
for both the scout and attack helicopter combined tasks 
(Tables 16 and 17) were generally poorer than those for 
the individual tasks (Tables 14 and 15). This indicates 
that the workload results obtained via both techniques 
for individual tasks correlated somewhat better than did 
those for the combined tasks. 


The significance level values in Tables 18 and 19 
indicate that the TLM's two workload measurement 
techniques do not correlate highly with each other. 


TABLE 18. RANK CORRELATION SIGNIFICANCE LEVELS OF TLM 
INDIVIDUAL TASK WORKLOAD VALUES 





TABLE 19. RANK CORRELATION SIGNIFICANCE LEVELS OF TLM 
COMBINED TA8K WORKLOAD VALUES 
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C. GRAPHIC ANALYSIS OF MODEL DATA AND SURVEY DATA 
1. Data Selection for Further Analysis 

Several graphic plots were prepared to make further 
comparisons between results from the two workload measuring 
devices. All of the survey data used for the graphic analysis 
came from the workload task ratings data, based on a scale of 
O* "to Tie With a minor transformation, these values were 
directly comparable to the TLM output, which was based on a 
scale of O to 100. Survey data used for the graphic 
comparisons included AHR (Table 10, top) for the attack 
helicopter individual tasks, SH-CV (Table 4, top) for the 
scout helicopter individual tasks, AH-AB (Table 4, bottom) for 
the attack helicopter combined tasks, and SHR (Table 10, 
bottom) for the scout helicopter combined tasks. 

The four task-rating data vectors utilized for the 
graphic comparisons (AHR, SH-CV, AH-AB, and SHR) have been 
shown to be robust and internally valid for use as baseline 
workload values (Chapter III, Section G). An additional 
criterion for the survey data used for the comparisons was 
that these vectors contained workload values from the greatest 
number of respondents for a given survey section. For 
example, the SH-CV vector (Table 4, top) was chosen over SH-AB 
(Table 4, top) because SH-CV represented responses from 12 
participants while SH-AB values were obtained from only six 
participants. 
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2. Results of Analysis 

The nine workload plots included in this section are 
based on a workload scale ranging from 0O to 100. Smaller 
values represent lower workload values. Survey data is 
plotted with a solid line in each graph. Model-generated data 
is plotted with a dashed line. Due to its demonstrated 
validity and robustness, the survey data has been taken as 
"ground truth" for these comparisons. 

The first plot, Figure 9, shows the only set of model- 
generated (SCT) and survey (SH-AB) workload values’ that 
correlated with a significance level below 0.05. DISst ince 
differences in workload across the tasks may be observed. The 
model data generally tracks values from the survey data; 
however, the model data's "flatness" implies that it may have 
poorer resolution in measuring workload levels. Nevertheless, 
considering the high data correlation, Figure 9 serves as a 
base from which to view the other graphs. 

The remaining eight graphs are characterized by the 
following observations: 


1. Figure 10 shows that the attack helicopter individual- 
task model data (ATK) agrees reasonably well with the 


associated survey data (AHR). Primary differences are 
for Task 1 (Follow scout) and Task 7 (Listen to radio 
transmission). 
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WORKLOAD 





4a 6 
INOIVIOUAL TASKS 


Figure 9. Highly Correlated Scout Helicopter Model (SCT) 
and Survey (SH-AB) Workload Values 


WORKLOAD 





~ 6 
INOTVIOQUAL TASKS 


Figure 10. Attack Helicopter Model (ATK) and Survey (AHR) 
Workload Values 


Figure 11 indicates that the TLM BSCATK data over- 
estimates workload and provides little resolution for 
attack helicopter individual tasks. 
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WORKLOAD 


— AHR 
-+ BSCATK 





4 6 
INDIVIDUAL TASKS 


Figure 11. Attack Helicopter Model (BSCATK) and survey 
(AHR) Workload Values 


Figure 12 is very similar to Figure 9. The survey 
data of Figure 12 comes from the Cavalry Squadron 
scout pilots (SH-CV). The survey data of Figure 9 comes 
from the Attack Battalion scout pilots (SH-AB). 
Comparing Figure 9 with Figure 12, it may be seen that 
the model provides a slightly better measure of Attack 
Battalion scout pilot workload than Cavalry Squadron 
workload. The major discrepancy is for Task 7 (Listen 
to radio transmission). 


WORKLOAD 





a 6 
INDIVIDUAL TASKS 


Figure 12. Scout Helicopter Model (8CT) and Survey (8H- 
CV) Workload Values 
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4. 


Figure 13 is similar to Figure 11. Again, the BSCSCT 
data overestimates workload and provides little 
resolution for scout helicopter individual tasks. 


WORKLOAD 





a =) 
INDIVIDUAL TASKS 


Figure 13. Scout Helicopter Model (BSCSCT) and Survey 
(SH-CV) Workload Values 


Figures 14 through 17 all involve combined tasks and 
display similar results. In each case the model- 
generated workload values are much higher than the survey 
values for most tasks. The model values for ATK (Figure 
14) and SCT (Figure 16) show some trend agreement with 
the survey values, but greatly overestimate all but the 
first and last tasks. The model's BSCATK (Figure 15) and 
BSCSCT (Figure 17) values show almost no resolution of 
workload levels over the six combined tasks. 


WORKLOAD 





8 1 = 3 2 5 =) 
COMBINED TASKS 


Figure 14. Attack Helicopter Model (ATK) and Survey (AH- 
AB) Workload Values 
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WORKLOAD 





3 
COMBINED TASKS 


Figure 15. Attack Helicopter Model (BSCATK) and Survey 
(AH-AB) Workload Values 


WORKLOAD 





3 
COMBINED TASKS 


Figure 16. Scout Helicopter Model (8CT) and Survey (8HR) 
Workload Values 


Do 


WORKLOAD 


-+ BSCSCT 





3 
COMBINED TASKS 


Figure 17. Scout Helicopter Model (BSCSCT) and Survey 
(SHR) Workload Values 
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V. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 
The purpose of this study was to determine the validity of 
the A°I TLM as a device to measure the workload levels 
experienced by Army helicopter pilots. Empirically-derived 
workload data was collected from Army pilots, analyzed, and 
correlated with workload measurements generated by the TLM. 
Analysis of the empirical data provided insights into pilot- 
generated workload levels and provided a_e statistically 
acceptable basis upon which to judge the accuracy of TLM 
outputs. Furthermore, insights into the opinions of Army 
helicopter pilots concerning workload provides direction for 
future refinement of the TLM. 
1. TLM Validity 
In its present configuration, output from the TLM does 
not consistently yield measurements of Army helicopter pilot 
workload that correlate with pilot opinions of their workload 
levels, for the set of tasks tested. Of the two TLM 
techniques of evaluating workload, the second technique should 
be abandoned. This technique adds task-generated workload 
onto constant workload resulting from a basic underlying 


flying task (BSCATK and BSCSCT). This technique showed little 


Do 


workload level resolution and overestimated workload values 
for almost every task. 

However, the first technique utilized by the TLM 
appears to provide better results. This technique analyzes 
aviator tasks separately from the actions associated with 
hands-on flying. Graphic analysis showed workload trends that 
in part agreed with the empirical measurements of workload. 
However, it should be noted that agreement between the model 
and the survey data was greater for the individual task 
measurements than for those related to the combined tasks. 

2. Army Helicopter Pilot Workload Perceptions 

Outputs from the TLM currently indicate that workload 
levels for the scout and attack helicopter pilots are almost 
identical. For essentially the same set of tasks embedded in 
realistic scout and attack helicopter scenarios, the data 
collected via the pilot surveys indicates that aviator 
workload varies according to the type aircraft flown and the 
type of mission conducted by the pilots. It is obvious that 
there are major discrepancies between the two methods for 
determining workload. 

Results of the Friedman test indicate that survey 
responses from the Attack Helicopter Battalion participants 


cannot be pooled with those from the Cavalry Squadron. This 
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is a strong indication that workload levels are not 
experienced the same by all of the aviators tested. The 
Spearman and Wilcoxon tests showed that, even among the scout 
pilots, workload is perceived differently by those scouts in 
the Cavalry Squadron than by those in the Attack Helicopter 
Battalion. These results provide strong evidence that the TLM 
may need modification to adjust workload according to the 
aviator's aircraft and specific mission. 

B. RECOMMENDATIONS 

Analysis of the degree of correlation between the TLM's 
workload output and the empirically-generated workload values 
obtained via the survey results in three recommendations. 
First, the TLM in its present configuration should not be used 
as the workload measuring mechanism in the MIDAS program. The 
TLM generally overestimates workload and provides an overly- 
narrow range of values for workload experienced by pilots 
performing a variety of tasks. 

Second, the TLM shows promising potential in the measuring 
of workload associated with individual helicopter pilot tasks. 
This aspect of the TLM should be enhanced to take maximum 
advantage of this capability. Initial enhancements should 
address the mission and aircraft involved in generating pilot 
workload. Although the survey data indicates that these 
factors are significant, the TLM generates generic workload 
measurements independent of them. 
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Third, empirical workload data collected as described in 
this thesis should be used for future evaluations of the TLM. 
As the TLM matures, efforts should be made to ensure that 
output correlates highly with workload data obtained directly 
from Army pilots, as described in this study. However, the 
limited range of the survey data reported here should not 
serve as the only means of validating the TLM. TLM output 
should also be compared to other empirically-derived workload 
measurements, and to the output of other models designed to 
measure pilot workload. 

C. SUGGESTIONS FOR FURTHER RESEARCH 

Several aspects of this study deserve further invest- 
igation. First, helicopter pilot workload data should be 
obtained from another, independent source. This data then 
could be compared with the empirically-derived data collected 
in this study, to insure data validity. This same independent 
workload data also could be compared to the TLM's output. 
Second, the survey results reported here may be based on 
factors not investigated in this study, such as individual 
pilot experience. A more detailed analysis of the individual 
pilot responses may be warranted. Third, expanded surveys 
Similar to those used for this study should be conducted in 


the future to broaden and refine the workload data base. 
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APPENDIX A. SCOUT HELICOPTER WORKLOAD SURVEY 


SURVEY ON HELICOPTER PILOTING WORKLOAD 


Thank you for your participation in this survey. Your 
assistance helps a NASA-Army team develop a human factors 
design tool aimed at improving the layout of future Army 
helicopter cockpits. 

There are no right or wrong answers to the questions in 
this survey. The intent is to determine relative workload 
that you experience when flying. Your responses will help us 
determine what tasks you feel are harder or easier than others 
for the given scenario. Please answer the questions candidly. 

The survey iS anonymous. The combined data from all of 
the surveys will produce an overall "pilot" profile of task 
ratings. 

The ratings and comparisons in the survey are listed 
randomly. No two surveys are exactly alike. 

If you have any questions, hold up your hand and the 


Survey administrating officer will assist you. 


PAGE 1 OF 12 


Do 


HOW TO FILL OUT THE SURVEY 


Evaluate the following maneuvers as an individual crewman 
in the context of the given scenario and helicopter terrain 
flight in general. Difficulty is defined as a relative measure 
of combined physical and mental effort. 


Evaluations are done in two parts. First, rate each 
maneuver by circling a value along the degree-of-difficulty 
continuum to indicate its difficulty. 


Example 1: 
MANEUVER DEGREE OF DIFFICULTY 
low high 
Fly straight and level O 1 2 3 4 «5 6 Fos See 
Autorotate at night O 1 2 3 4 556 7.58) Soa 


For the second part of the survey, individual maneuvers 
are compared with each other. 


Choose the more difficult of the two by filling in the 


circle next to it. 


Example 2: 


Fly straight and level Oo ae Autorotate at night 


PAGE 2 OF 12 


60 


SCOUT HELICOPTER SCENARIO 


SETTING: You are the pilot of an OH-58 conducting a route 
reconnaissance in an unfamiliar hilly/wooded area. You are in 
the terrain flight mode along one side of the route, clearing 
the area while observing the route as frequently as possible. 
Due to pilot shortages, you are accompanied by a crewchief 
sitting in the left seat. He is familiar with the cockpit. 
He can hold a 1:50,000 map for you to observe. However, he 
cannot fly and you must double check his map reading 
activities. You are flying during the day with clear and calm 
weather conditions. An attack helicopter is overwatching you. 
However, you will not communicate with him during this mission 


segment. Enemy contact is possible. 
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ot 


Rate the difficulty of each maneuver by circling a value along 


the degree of difficulty continuum. 


MANEUVER DEGREE OF 


low 


DIFFICULTY 


high 


6 7 8.355 


6 7 8m 


ee ce ee ee 


Tell crewchief "Unmasking" 0 1 2 3 4 5 


UnmaskaircraftO 1 2 3 4 5 


Listen to crewchief describe what he sees 0 1 2 3 4 § 


Turn up volume on ICS control panel O 1 2 3 4 #57 


Determine present location (6 digit grid) 0 1 2 3 4 §5 


as crewchief holds map 


SURVEY # OHITR 1 
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6.7 Seca 


6 7 8 oan 


6 7 8 Jee 


6 7 6 “Saat 


6 7 8 ae 
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Rate the difficulty of each maneuver by circling a value along 


the degree of difficulty continuum. 


MANEUVER 


NOE deceleration behind another scout 0 


Check torque meter and announce reading 0 
while NOE decelerating 


Check torque meter and announce reading 0 
while hovering 


Tell crewchief "Unmasking" while unmasking 0 
aircraft 


Listen to crewchief describe what he sees 0 
while you turn up volume on ICS control 
panel and hover 


Determine present location (6 digit grid) 0O 
as crewchief holds map and you listen 
to him describe what he sees 


SURVEY # OHCTR 1 
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DEGREE OF DIFFICULTY 


low high 


10 


10 


10 


10 


10 
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Compare the two maneuvers 


aifficult of “Ghe two by 


friling 


MANEUVER 1 


Follow 


Follow 


Follow 


Follow 


Follow 


Follow 


Follow 


Follow 


NOE 


NOE 


NOE 


NOE 


SURVEY # OHIPC 1-1 


another 


another 


another 


another 


another 


another 


another 


another 


on the same line. Choose the more 
in the circle next to it. 


scout O 


scout O 


scout O 


scout O 


scout O 


scout O 


scout O 


scout O 


deceleration O 


deceleration O 


deceleration O 


deceleration O 
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MANEUVER 2 


NOE deceleration 


Check torque meter and announce 
reading 


Hover 


Tell crewchief "Unmasking" 


Unmask aircraft 


Listen to crewchief describe what 
he sees 


O Turn up volume on ICS control panel 


O Determine present location (6 digit 


grid) as crewchief holds map 


Check torque meter and announce 
reading 


Hover 


Tell crewchief "Unmasking" 


Unmask aircraft 
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NOE 


NOE 


NOE 


Check torque meter 


Check torque meter 


Check torque meter 


Check torque meter 


Check torque meter 


Check torque meter 


SURVEY # OHIPC 1-2 


MANEUVER 1 


MANEUVER 2 


deceleration O O Listen to crewchief describe what 


he sees 


deceleration O O Turn up volume on ICS control panel 


deceleration O O Determine present location (6 digit 


and announce 
reading 


and announce 
reading 


and announce 
reading 


and announce 
reading 


O 


O 


grid) as crewchief holds map 


O Hover 


O Tell crewchief "Unmasking" 


O Unmask aircraft 


O Listen to crewchief describe what 
he sees 


and announce O O Turn up volume on ICS control panel 


reading 


and announce O O Determine present location (6 digit 


reading 


grid) as crewchief holds map 


Hover O O Tell crewchief "Unmasking" 


Hover O oO Unmask aircraft 
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MANEUVER 1 MANEUVER 2 


Hover O O Listen to crewchief describe what 
he sees ie 


Hover O O Turn up volume on ICS control panel 


Hover O O Determine present location (6 digit 
grid) as crewchief holds map 


Tell crewchief "Unmasking" O O Unmask aircraft 


Tell crewchief "Unmasking" O O Listen to crewchief describe what 
he sees 


Tell crewchief "Unmasking" O O Turn up volume on ICS control panel 


Tell crewchief "Unmasking" O O Determine present location (6 digit 
grid) as crewchief holds map 


Unmask aircraft O O Listen to crewchief describe what 
he sees 


Unmask aircraft O O Turn up volume on ICS control panel 


Unmask aircraft O O Determine present location (6 digit 
grid) as crewchief holds map 


Listen to crewchief describe what O O Turn up volume on ICS control panel 
he sees 


SURVEY # OHIPC 1-3 PAGE 8 OF 12 
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MANEUVER 1 MANEUVER 2 


Listen to crewchief describe what O O Determine present location (6 digit 
he sees grid) as crewchief holds map 


Turn up volume on ICS control panel O O Determine present location (6 digit 
grid) as crewchief holds map 


SURVEY # OHIPC 1-4 PAGE 9 OF 12 
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Compare the two maneuvers on the same line. Choose the more 
difficult of the two by filling in the circle next to it. 


MANEUVER 1 


NOE deceleration behind another O O 


SCcoue 


NOE deceleration behind another O O 


scout 


NOE deceleration behind another O O 


scout 


NOE deceleration behind another O O 


scout 


MANEUVER 2 


Check torque meter and announce 
reading while NOE decelerating 


Check torque meter and announce 
reading while hovering 


Tell crewchief "Unmasking" while 
unmasking aircraft 


Listen to crewchief describe what 
he sees while you turn up volume 
on ICS control panel and hover 


NOE deceleration behind another O O Determine present location (6 digit 


Check torque meter 
reading while NOE 


Check torque meter 
reading while NOE 


Check torque meter 
reading while NOE 


SURVEY # OHCPC 1-1 


scout 


and announce O O 
decelerating 


and announce O O 
decelerating 


and announce O O 
decelerating 
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grid) as crewchief holds map and 
you listen to him describe what 
he sees 


Check torque meter and announce 
reading while hovering 


Tell crewchief "Unmasking" while 
unmasking aircraft 


Listen to crewchief describe what 
he sees while you turn up volume 
on ICS control panel and hover 
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MANEUVER 1 MANEUVER 2 


Check torque meter and announce O O Determine present location (6 digit 


reading while NOE decelerating grid) as crewchief holds map and 
you listen to him describe what 
he sees 


Check torque meter and announce O O Tell crewchief "Unmasking" while 
reading while hovering unmasking aircraft 


Check torque meter and announce O O Listen to crewchief describe what 
reading while hovering he sees while you turn up volume 
on ICS control panel and hover 


Check torque meter and announce O O Determine present location (6 digit 


reading while hovering grid) as crewchief holds map and 
you listen to him describe what 
he sees 


Tell crewchief "Unmasking" while O O Listen to crewchief describe what 
unmasking aircraft he sees while you turn up volume 
on ICS control panel and hover 


Tell crewchief "Unmasking" while O O Determine present location (6 digit 


unmasking aircraft grid) as crewchief holds map and 
you listen to him describe what 
he sees 


Listen to crewchief describe what O O Determine present location (6 digit 


he sees while you turn up volume grid) as crewchief holds map and 
on ICS control panel and hover you listen to him describe what 
he sees 
SURVEY # OHCPC 1-2 PAGE 11 OF 12 
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PILOT BACKGROUND INFORMATION 


Please answer the following questions regarding your 
aviation experience and qualifications. 


ie 


Approximately how many 
the following aircraft 
(Leave blank if little 
OH-58A/C 

OH-58D 

UH-1 


UH-60 


flight hours do you have in 
as either a PC or PI? 
or no experience) 

AH-1 

AH-64 


other rotary wing 


fixed wing 


Answer the following by indicating the approximate 


number of flight hours. 


a) Your total flight tine: 


b) Flight time as an Instructor Pilot: 


c) Combat flight time: 


dad) Flight time while qualified as a PC: 


Answer the following by indicating the approximate 


number of years. 


a) Years as a PC: 


b) Years as an Army Aviator: 


END OF SURVEY 


THANK YOU FOR YOUR TIME AND EFFORT 
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APPENDIX B. ATTACK HELICOPTER WORKLOAD SURVEY 


SURVEY ON HELICOPTER PILOTING WORKLOAD 


Thank you for your participation in this survey. Your 
assistance helps a NASA-Army team develop a human factors 
design tool aimed at improving the layout of future Army 
helicopter cockpits. 

There are no right or wrong answers to the questions in 
this survey. The intent is to determine relative workload 
that you experience when flying. Your responses will help us 
determine what tasks you feel are harder or easier than others 
for the given scenario. Please answer the questions candidly. 

The survey is anonymous. The combined data from all of 
the surveys will produce an overall "pilot" profile of task 
ratings. 

The ratings and comparisons in the survey are listed 
randomly. No two surveys are exactly alike. 

If you have any questions, hold up your hand and the 


Survey administrating officer will assist you. 


PAGE 1 OF 12 


di 


HOW TO FILL OUT THE SURVEY 


Evaluate the following maneuvers as an individual crewman 
in the context of the given scenario and helicopter terrain 
flight in general. Difficulty is defined as a relative measure 
of combined physical and mental effort. 


Evaluations are done in two parts. First, rate each 
maneuver by circling a value along the degree-of-difficulty 
GCOncinuum to indicate 1tsecdit ticul ty. 


Example 1: 
MANEUVER DEGREE OF DIFFICULTY 
low high 
Fly straight and level O 2 2 3 4 5 6 J 8a 
Autorotate at night O 1 2 3 4 5 6”™7 8 ome 


For the second part of the survey, individual maneuvers 
are compared with each other. 


Choose the more difficult of the two by filling in the 


Circle next to it. 


Example 2: 


Fly straight and level om. Autorotate at night 
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ATTACK HELICOPTER SCENARIO 


SETTING: You are the pilot of an armed AH-1 flying in the 
terrain flight mode. You are conducting travelling overwatch 
by following an aeroscout through an unfamiliar hilly/wooded 
area to a battle position. You will receive a target handoff 
from the firing position. Your crewmember will prepare fora 
TOW missile engagement and will not fly the aircraft during 
this mission segment. You are flying during the day with 


clear and calm weather conditions. Enemy contact is possible. 


PASE. 3, OF AZ 


we3 


Rate the difficulty of each maneuver by circling a value along 
the degree of difficulty continuum. 


MANEUVER DEGREE OF DIFFICULTY 


low high 


FollowscoutO 1 2 3 4 5 6 7 )33 

NOEdecelerationO 1 2 3 4 5 6 7 6a 

Check torque meter and announce reading 0_1 +2 #3 +4 #5 6 7 +8 9 _ 10 

Hover O 1 2 3 4 5 6 7 Sea 

Tell crewmember "Unmasking" 0 1 #2 #3 #4 5 6 7 8 9 _ 10 

UnmaskaircraftO 1 2 3 4 5 6 #7 @3333 

Listen to radio transmission from scout 0 1 2 3 4 5 6 7 S323 
(direction to fire, range, target) 

Switch master armto"“arm” 0 1 2 3 4 5 6 7 83a 

Interpret gunners position of TSU basedon 0 1 2 3 4 5 6 7 8 9 10 
PSI information 

SURVEY # AHITR 1 PAGE 4 OF 12 
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Rate the difficulty of each maneuver by circling a value along 
the degree of difficulty continuum. 


MANEUVER 


NOE deceleration behind scout O 


Check torque meter and announce reading O 


Check torque meter and announce reading O 


Tell crewmember "Unmasking" while unmasking O 


Listen to radio transmission from scout O 
(direction to fire, range, target) while 


Interpret gunners position of TSU, based 0 


while NOE decelerating 


while hovering 


aircrart 


DEGREE OF DIFFICULTY 


low high 


dee 20S a ee) Oe ee 


tS ot Ae OC SEP eG eo SO 


1, 2k 0 AG oe i Oe 9 VIO 


je ae ee SO ie ees, Be oe ee, 


ec ie ee 


Dy eed) 4. BONS 1 an Oe SD 


Switching master arm to "arm" 


on PSI information, while receiving 


SURVEY # AHCTR 


1 


target handoff from scout 


f= 


Dea HS 34 ee 17 Oy 3 1D 


PAGE 5 OF 12 


Compare the two maneuvers 
difficult of the -two by 77 2ling 


MANEUVER 1 


Follow 


Follow 


Follow 


Follow 


Follow 


Follow 


Follow 


Follow 


on the same line. Choose the more 


in the circle next to it. 


MANEUVER 2 


scout O O NOE deceleration 


scout O O Check torque meter and announce 
reading 


scout O O Hover 


scout O O Tell crewmember "Unmasking" 


scout O O Unmask aircraft 


scout O O Listen to radio transmission from 
scout (target, range, direction) 


scout O O Switch master arm to "arm" 


scout O O Interpret gunners position of TSU 
based on PSI information 


NOE deceleration O O Check torque meter and announce 


reading 


NOE deceleration O O Hover 


NOE deceleration O O Tell crewmember "Unmasking" 


SURVEY # AHIPC 1-1 
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MANEUVER 1 MANEUVER 2 


NOE 


NOE 


NOE 


NOE 


Check torque meter 


Check torque meter 


Check torque meter 


deceleration O O Unmask aircraft 


deceleration O O Listen to radio transmission from 
scout (target, range, direction) 


deceleration O O Switch master arm to "arm" 


deceleration O O Interpret gunners position of TSU 
based on PSI information 


and announce O O Hover 
reading 


and announce O O Tell crewmember "Unmasking" 
reading 


and announce O O Unmask aircraft 
reading 


Check torque meter and announce O O Listen to radio transmission from 


Check torque meter 


reading scout (target, range, direction) 


and announce O O Switch master arm to "arm" 
reading 


Check torque meter and announce O O Interpret gunners position of TSU 


SURVEY # AHIPC 1-2 


reading based on PSI information 





Hover O O Tell crewmember "Unmasking" 
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MANEUVER 1 MANEUVER 2 





Hover O O Unmask aircraft 


Hover O O Listen to radio transmission from 
scout (target, range, direction) 


Hover O O Switch master arm to "arm" 


Hover O O Interpret gunners position of TSU 
based on PSI information 


Tell crewmember "Unmasking" O O Unmask aircraft 


Tell crewmember "Unmasking" O O Listen to radio transmission from 
scout (target, range, direction) 


Tell crewmember "Unmasking" O O Switch master arm to "arm" 


Tell crewmember "Unmasking" O O Interpret gunners position of TSU 
based on PSI information 


Unmask aircraft O O Listen to radio transmission from 
scout (target, range, direction) 


Unmask aircraft O O Switch master arm to "arm" 


Unmask aircraft O O Interpret gunners position of TSU 
based on PSI information 


Listen to radio transmission from O O Switch master arm to "arm" 
scout (target, range, direction) 
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MANEUVER 1 MANEUVER 2 


Listen to radio transmission fromO O Interpret gunners position of TSU 
scout (target, range, direction) based on PSI information 


Switch master arm to "arm" O O Interpret gunners position of TSU 
based on PSI information 
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fie) 


Compare the two maneuvers on the same line. Choose the more 
difficult of the two by filling in the circle next to it. 


MANEUVER 1 MANEUVER 2 


NOE deceleration behind scout O O Check torque meter and announce 
reading while NOE decelerating 


NOE deceleration behind scout O O Check torque meter and announce 
reading while hovering 


NOE deceleration behind scout O O Tell crewmember "Unmasking" while 
unmasking aircraft 


NOE deceleration behind scout O O Listen to radio xsmn from scout 
(direction, range, target) while 
switching master arm to "arm" 


NOE deceleration behind scout O O Interpret gunners position of TSU 
based on PSI info, while 
receiving target handoff from 
scout 


Check torque meter and announce O O Check torque meter and announce 
reading while NOE decelerating reading while hovering 


Check torque meter and announce O O Tell crewmember "Unmasking" while 
reading while NOE decelerating unmasking aircraft 


Check torque meter and announce O O Listen to radio xsmn from scout 
reading while NOE decelerating (direction, range, target) while 
switching master arm to “arm" 


Check torque meter and announce O O Interpret gunners position of TSU 


reading while NOE decelerating based on PSI info, while 
receiving target handoff from 
scout 
SURVEY # AHCPC 1-1 PAGE 10 OF 12 
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MANEUVER 1 MANEUVER 2 


Check torque meter and announce O O Tell crewmember "Unmasking" while 
reading while hovering unmasking aircraft 


Check torque meter and announce O O Listen to radio xsmn from scout 
reading while hovering (direction, range, target) while 
switching master arm to "arm" 


Check torque meter and announce O O Interpret gunners position of TSU 


reading while hovering based on PSI info, while 
receiving target handoff from 
scout 


Tell crewmember "Unmasking" while O O Listen to radio xsmn from scout 
unmasking aircraft (direction, range, target) while 
switching master arm to "arm" 


Tell crewmember "Unmasking" while O O Interpret gunners position of TSU 


unmasking aircraft based on PSI info, while 
receiving target handoff from 
scout 


Listen to radio xsmn from scout O O Interpret gunners position of TSU 


(direction, range, target) while based on PSI info, while 
Switching master arm to "arm" receiving target handoff from 
scout 
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PILOT BACKGROUND 


INFORMATION 


Please answer the following questions regarding your 
aviation experience and qualifications. 


1. 


Approximately how many 
the following aircraft 
(Leave blank if little 
OH-58A/C 

OH-58D 

Wns i 


UH-60 


flight hours do you have in 
as either a PC or PI? 
or no experience) 

AH-1 

AH-64 


other rotary wing 


fixed wing 


Answer the following by indicating the approximate 


number of flight hours. 


a) Your total flight time: 


b) Flight time as an Instructor Pilot: 


c) Combat flight time: 


ad) Flight time while qualified as a PC: 


Answer the following by indicating the approximate 


number of years. 


a) Years as a PC: 


b) Years as an Army Aviator: 


END OF SURVEY 


THANK YOU FOR YOUR TIME AND EFFORT 
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APPENDIX C. SPEARMAN RANK CORRELATION SIGNIFICANCE LEVELS 


INDIVIDUAL-TASK PAIRED COMPARISONS 


pt Tf 079 | 0.069 | = 


0.027 
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COMBINED-TASK PAIRED COMPARISONS 


_ | an-ap | an-cv | s-ap | su-cv | go-ap | zo-cv | 


/ | Tf 00 | 0.000 | = 


} 0.060 


EEE 


COMBINED-TASK RATINGS 
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